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ABSTRACT 

Given that few studies have formally examined pedagogical design considerations of Massive 
Online Open Courses (MOOCs), this study explored variations in the pedagogical design of six 
MOOCs offered at the University of Toronto, while considering disciplinary characteristics and 
expectations of each MOOC. Using a framework (Neumann et ah, 2002) characterizing 
teaching and learning across categories of disciplines, three of the MOOCs represented social 
sciences and humanities, or “soft” MOOCs, while another three represented sciences, or “hard” 
MOOCS. We utilized a multicase study design for understanding differences and similarities 
across MOOCs regarding learning outcomes, assessment methods, interaction design, and 
curricular content. MOOC instructor interviews, MOOC curricular documents, and discussion 
forum data comprised the data set. Learning outcomes of the six MOOCs reflected broad 
cognitive competencies promoted in each MOOC, with the structure of curricular content 
following disciplinary expectations. The instructors of soft MOOCs adopted a spiral curriculum 
and created new content in response to learner contributions. Assessment methods in each 
MOOC aligned well with stated learning outcomes. In soft MOOCs, discussion and exposure to 
diverse perspectives were promoted while in hard MOOCs there was more emphasis on 
question and answer. This study shows disciplinary-informed variations in MOOC pedagogy, 
and highlights instructors’ strategies to foster disciplinary ways of knowing, skills, and practices 
within the parameters of a generic MOOC platform. Pedagogical approaches such as peer 
assessment bridged the disciplines. Suggestions for advancing research and practice related to 
MOOC pedagogy are also included. 
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PURPOSE 

Few studies have formally examined pedagogical design considerations of MOOCs. In this study 
we focused on deconstructing curriculum design and implementation in six MOOCs representing a 
range of disciplinary affiliation and then identified promising practices. The purpose of this research is to 
understand Massive Open Online Course (MOOC) design across disciplines with respect to four 
dimensions: learning outcomes, assessment of learning outcomes, interaction design, and curricular 
content (Neumann, Parry, & Becher, 2002). Within existing literature, students' overall learning 
experience, course design, and course implementation in higher education may reflect discipline-specific 
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epistemological differences. Researchers have pointed out such disciplinary differences in various 
dimensions of teaching practices, including fostering deep approaches to learning (Laird, Shoup, Kuh, & 
Schwarz, 2008); teacher vs. learner centered instructional approaches (Lindblom-Ylanne, Trigwell, 
Nevgi, & Ashwin, 2006); and assessment approaches (jessop & Maleckar, 2016). In the context of the 
University of Toronto (U of T), findings of a yearly report (Open UToronto MOOC Initiative: Report 
on Second Year of Activity, 2014) indicated differences in course design across U ofT MOOCs focused 
on different disciplines. Computer programming courses, for example, contained more automatically- 
graded assignments compared to courses in social work or aboriginal education. Motivated by these 
indicators and informed by existing research, this multicase study (Stake, 2013) examines variations in 
course design among six Coursera MOOCs from different disciplines offered by U of T. As a common 
ground for comparison across multiple MOOCs, we use an existing framework (e.g. Neumann et ah, 
2002) that characterizes teaching and learning practices in different disciplines. The framework adapted 
here is by no means restrictive and prescriptive of how pedagogical design of a course in a given 
discipline should be approached, but rather, provides a useful lens for understanding and discussion 
across disciplines. 

CONCEPTUAL FRAMEWORK 

Course design and implementation in higher education may reflect discipline specific 
epistemological differences in various dimensions of teaching practices, including teacher vs. learner 
centered instructional approaches (Lindblom-Ylanne et ah, 2006), assessment methods (jessop & 
Maleckar, 2016), and interaction design (Barbera, Layne, & Gunawardena, 2014). To guide our analysis 
of course design, we used a disciplinary categorization framework (Neumann et ah, 2002), derived from 
Biglan's (1973) disciplinary typology. Biglan (1973) distinguished epistemological differences between 
high and low paradigm disciplines with high paradigm disciplines having consensus over knowledge base 
and the methods of inquiry, whereas low paradigm disciplines embrace iterative cycles of knowledge 
construction. Rather than being restrictive and prescriptive, this framework offered a set of dimensions 
related to teaching and learning practices that facilitate the comparison of course design in MOOCs. 

Pedagogical Practices in Disciplinary Context 

Neumann et al. (2002) and Neumann (2003) proposed a framework with two dimensions of 
disciplinary knowledge and social practices to characterize teaching and learning across four categories 
of disciplines: hard pure (e.g. natural sciences), soft pure (e.g. humanities), hard applied (e.g. 
engineering), and soft applied (e.g. education). Three knowledge-related dimensions of Neumann’s 
(2003) framework, briefly described below, were relevant to this study: 

• Content and curriculum. Hard pure fields present structured and cumulative curricular 
content. Hard applied fields incorporate the application of knowledge and theories to 
practice. Curriculum in soft pure fields is expansive. Soft applied fields emphasize 
synthesis of ideas while considering their application in a professional context. 

• Cognitive processes and competencies. Students develop field-specific knowledge in 
hard pure fields. Reasoning skills and ability to use discipline-specific methods to judge 
the accuracy of data are other desired student outcomes. Students in soft fields are 
expected to develop argumentation and critical thinking skills. 

• Assessment. Hard pure fields incorporate frequent examinations to gauge students’ 
knowledge. Problems, in hard applied fields are another form of assessment. Soft pure 
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fields often incorporate analytical and argumentative essays. Formative assessment is 
common in soft fields. Soft applied fields include practice-related assessments. 

While disciplinary differences may lead to variations in teaching approaches, trans-disciplinary 
pedagogical guidelines exist that promote learning for understanding. For example, incorporating 
formative and frequent assessments with opportunities for peer- and self-assessment has been widely 
recommended (Cox, Imrie, Miller, & Miller, 2014). 

Variations in Teaching Approaches 

Previous studies have attended to differences in approaches to teaching and learning across 
disciplines and the impact of such difference on students’ learning experiences (Barbera et al., 2014; 
Lindblom-Ylanne et al., 2006; Norton, Richardson, Hartley, Newstead, & Mayes, 2005; Postareff, 
Virtanen, Katajavuori, & Lindblom-Ylanne, 2012). Here, we provide a brief review of selected studies to 
showcase such variations in approaches to teaching and learning, assessment, and interaction in courses 
across disciplines. 

To understand potential relations between disciplinary background and university instructors’ 
approaches to teaching, Lindblom-Ylanne et al. (2006) invited 340 instructors from two countries who 
represented hard and soft disciplinary categories, to complete a 16-item Approaches to Teaching 
Inventory (Trigwell & Prosser, 2004). The survey contained items addressing two categories of 
approaches to teaching: conceptual change/student-focused (CCSF), and information 
transmission/teacher-focused (ITTF). Within the 303 completed surveys, significant differences were 
observed on CCSF items among disciplinary categories, with hard category scoring lower in CCSF than 
soft disciplinary categories. As for ITTF items, instructors of hard disciplines scored significantly higher 
in these items than instructors of soft disciplines (Lindblom-Ylanne et al., 2006). Comparable findings 
are reported for more than 10,000 faculty members’ use of deep approaches to learning in their 
undergraduate teaching as expressed in a Faculty Survey of Student Engagement (Laird et al., 2008), 
concluding that faculty members in soft disciplines were more likely to engage their students in 
opportunities that involved deep approaches to learning. 

Regarding assessment methods, in a single discipline study, 28 instructors from a faculty of 
Pharmacy were asked about their beliefs regarding the purposes that assessment would serve and about 
their actual practice of classroom assessment (Postareff et al., 2012). The majority of the instructors 
expressed a knowledge reproduction conception of assessment and incorporated a limited range of 
summative assessment approaches in their practice. Postareff et al. (2012) also noticed an absence of 
self-assessment and rare use of peer-assessment. Dominant disposition and practice of assessment to 
measure knowledge reproduction was attributed to the characteristics of disciplinary knowledge in the 
field of Pharmacy. Different findings were reported, however, in a study that compared the variety of 
formative and summative assessment in three disciplinary groups of sciences, humanities, and 
professional programs (jessop & Maleckar, 2016). In their study, formative assessment was defined as a 
non-graded mandatory assessment task. Drawing on program audit data, the authors noticed that the 
number of formative assessment approaches in sciences courses was three times the number of 
formative assessments in humanities and professional programs disciplinary groups. They attributed 
such difference to the application of formative assessment in sciences to help students master the 
cumulative knowledge base of their degree program (jessop & Maleckar, 2016). 

According to the Community of Inquiry (Garrison, Anderson, & Archer, 2010) model, 
interaction in online courses manifests in at least three modes: student-content; student-instructor, and 
student-student. Studies that investigate disciplinary differences in integrating interaction have 
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considered either a single mode or multiple modes of interaction in online courses. To probe the level of 
cognitive, instructional, and social presence in discussion forums of sciences and humanities courses 
offered in an Open University, Gorsky, Caspi, Antonovsky, Blau, and Mansur (2010) compared 25 
discussion forums from science courses with 25 forums from humanities courses. Regarding the three 
dimensions of presence as defined by the Community of Inquiry model, humanities instructors were 
more active in the social presence dimension while science instructors showed higher levels of cognitive 
presence. Within the cognitive presence dimension, Gorsky et al. (2010) reported that science 
instructors posted more exploration messages compared with a higher number of triggering event posts 
sent by humanities instructors. The effect of instructional presence on learner participation in online 
discussions, however, is debated in online learning literature. 

While differences in various pedagogical components of course design and implementation have 
been highlighted in terms of disciplinary context (Arbaugh, Bangert, & Cleveland-Innes, 2010; Pace, & 
Middendorf, 2004), the process of course design on its own is understudied within higher education 
(Sharpe & Oliver, 2013). Such research is also missing in the case of pedagogical design of Massively 
Open Online Courses (MOOCs). Note that, in this study, we do not explore design issues related to 
MOOCs developed according to a connectivist approach to learning that highly emphasizes learning 
through social connections (Kop, Fournier, & Mak, 2011). Rather, we focus on MOOCs that are more 
closely modelled on typical higher education courses. 

MOOC Pedagogy 

At this time, formal studies of disciplinary variations in MOOC design that allow for comparison 
and contrast of findings are scarce. In one study, Swan, Bogle, Day, van Prooyen, and Richardson (2014) 
developed a rubric to characterize MOOC development along 10 dimensions, including epistemology, 
teacher/student centeredness, and cognitive focus of learning activities. Applying the rubric to 13 STEM 
(Science, Technology, Math, & Engineering) and four non-STEM MOOCs, Swan et al. (2014) 
observed differences in pedagogical approaches of the two categories of MOOCs: non-STEM courses 
were more student-centered and included activities that facilitated knowledge construction, rather than 
knowledge reproduction. 

As for interaction, Chandrasekaran, Ragupathi, Kan, and Tan (2015) examined ways to 
maximize the benefits of instructor intervention in MOOC discussion forums. In 61 Coursera MOOCs 
from various disciplines, Chandrasekaran et al. (2015) observed an overall higher proportion of 
instructor intervention in exam-related threads compared with lecture-related threads. With expected 
variations in the design of MOOCs, Chandrasekaran et al. (2015) have not explained where instructors' 
efforts in discussion forums were concentrated, in relation to learning outcomes and disciplinary 
expectations. 

Considering discipline-related variations across several dimensions of teaching practices, and 
the emergence of studies that examine pedagogical characteristics of MOOCs, further investigation of 
MOOC design process is compelling. The research question guiding this study is: in six Uof TMOOCs 
from different disciplines, how can course design be characterized in terms of learning outcomes, 
curriculum and content, assessment methods, and interaction design? 

STUDY DESIGN 

Using a multicase study design (Stake, 2006), we examined how U of T instructors designed six 

MOOCs from hard and soft disciplines. Case study design has been recognized as a conducive research 

strategy for Scholarship of Teaching and Learning (SoTL) inquiry (Pearson, Albon, & Hubball, 2015). 
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We used the hard/soft dimension of Neumann and colleagues’ (2002) disciplinary categorization. Note 
that at the time when these courses were designed and delivered, Coursera, the platform used for the six 
MOOCs, provided a universal template for course design which may account for some similarities in 
course components across the six MOOCs that we present and discuss in the rest of this paper. Guided 
by the template, each course included content, mostly in the form of lecture videos, assessment, and 
discussions. The six MOOCs, identified below by a shortened version of their original titles, provided a 
comprehensive set to study pedagogical practices in MOOC design within this particular university 
context. The total number oflearners enrolled in each MOOC as of October 2015 is indicated following 
the shortened title. 

• Soft MOOCs: AboriginalEd (20,966), IntroPsych (77,608), and MentalHealth 
(23,491) 

• Hard MOOCs: Programmingl (80,000), Programming2 (53,974), and Stats (48,687) 
We collected the following qualitative and quantitative data: (l) MOOC documents in 

Coursera (syllabus and course content); (2) Interviews with MOOC instructors; and, (3) MOOC 
discussion forum data. Each case was first analyzed individually. Afterwards, we conducted cross-case 
analysis by combining findings from each case relevant to common themes. 

FINDINGS 

Findings are presented in four sections: learning outcomes, curricular content, assessment 
methods, and interaction design. 

Learning Outcomes 

Each MOOC identified distinct learning outcomes, summarized below. 

• Soft MOOCs 

o AboriginalEd: deepening understanding of aboriginal worldviews; critical thinking; and 
using evidence to construct arguments 

o IntroPsych: gaining foundational knowledge on various topics in psychology; critical 
thinking; active learning; and argumentation 
o MentalHealth: understanding historical background of the field; understanding the 
impact of contextual social/cultural issues affecting dispositions towards mental health 
and treatment options; and developing anti-stigma awareness 

• Hard MOOCs 

o Stats: gaining foundational knowledge of statistical methods and their application; and 
understanding strengths and weaknesses of statistical methods in context 
o Programmingl and 2: Programmingl, addressing computational thinking and 

fundamental concepts of computer programming. Programming2, focusing on style and 
efficiency of code 

Two soft MOOCs-IntroPsych and AboriginalEd-explicitly emphasized critical thinking, argument 
development, and knowledge construction. These competencies can be applied beyond a single course 
or degree program. A main goal in AboriginalEd MOOC was to engage learners with a range of 
knowledge about aboriginal worldviews by delegating cognitive responsibility to MOOC learners. The 
course instructor explained: 
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Saying this is how we are going to go about learning and understanding as a kind of a Journey, 
then, they [learners] become responsible for ensuring that they are constantly pushing 
themselves. I diditmore to help those who already know a lot so they are not getting bored, 
(instructorinterview, AboriginalEd) 

Learning outcomes of Hard MOOCs were more concrete regarding discipline-related 
competencies and skills that the learner would gain by completing the course. Yet critical thinking was 
not excluded from hard MOOCs. The Stats instructor, for example, explained: 

Id much rather that students, rather than just end up with a cookbook, you know, I got this set 
of ingredients, so I look up a recipe’... I’d rather them get some critical thinking skills about 
howthese things are used and what they are good for. (Instructor interview, Stats) 

Curriculum and Content 

All six MOOCs mainly presented content via lecture videos, as prompted by the course design 
template provided in the Coursera platform. We examined the average length and the number of 
lectures in each week for all sixMOOCs. While the number of weekly lectures was comparable across 
courses, two soft MOOCs, AboriginalEd and IntroPsych, on average had the longest total lecture time, 
106 minutes and 146 minutes respectively, exceeding the average length of lectures in other MOOCs by 
at least 30 minutes. 

Stats, AboriginalEd, and Intropsych MOOCs included optional videos: R statistical package 
tutorials, Screenside chats, and Side Dishes, respectively. R statistical package tutorial videos were 
included in course material from the beginning of the Stats MOOC and were mentioned in the “About 
the Course” page. Screenside chats and Side Dishes were conceptualized and created while the 
AboriginalEd and the IntroPsych MOOCs were offered. Document analysis and instructor interview 
data revealed that AboriginalEd, MentalHeath, and IntroPsych MOOC instructors expanded their 
curricular content through these additional videos. The videos responded to issues and questions that 
evolved through learners’ engagement with course content and their contributions to discussion forums. 

According to the IntroPsych instructor, making the instructor's reaction to the ongoing course 
discussions visible by publishing Side Dish videos, allowed learners who were not engaged in course 
discussions to become aware of important topics that had emerged from their fellow learners’ 
contributions. Another approach, taken by the MentalHealth MOOC instructor, was to provide weekly 
commentary of discussions and lessons learned in the previous week. 

Curricular content of soft MOOCs shared one characteristic: all three MOOCs complemented 
their lectures, presented by the MOOC instructor, with outside resources, including interviews with 
experts, other videos, books, and blog posts. The highest number of external resources belonged to the 
IntroPsych course with 168 external videos and 125 external online resources. 

Instructors of soft MOOCs encouraged students to embrace different or contradictory 
viewpoints regarding course topics. An expansive curriculum allowed students to use course topics as a 
starting point and “challenge themselves to go deeper” (instructor interview, AboriginalEd) if curricular 
content was already familiar to them. Exposing learners to contextual differences in viewpoints was a 
goal echoed by the MentalHealth instructor, given that the course included material from different 
countries. 

Hard MOOCs included fewer external resources and adopted a more linear approach to 

curriculum design. When asked about the likeliness of students in introductory statistics courses facing 
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opposing perspectives, the Stats MOOC instructor explained: “I don’t think that there is that much 
controversy at this level. Where there is controversy, it is too advanced.” 

The two programming MOOCs were taught by the same instructors and shared a suggested 
textbook, links to e-books, and three websites. While external resources were fewer in quantity 
compared to soft MOOCs, students in Stats, Programmingl, and Programming! courses shared their 
lecture notes or codes in wiki spaces, so that other students could use the material if needed. Instructors 
in the Programmingl course encouraged learners to contribute to the course wiki: 

If you've recently solved a tough installation issue or gotten helpful advice about a programming 

exercise or the video lectures or some other facet of the course > please take some time to help out 

your fellow classmates by writing about it. Feel free to create pages, clarify answers, and 

otherwise take ownership of these pages (Programmingl, Course wiki). 

Overall, curricular content and resources in these six hard and soft MOOCs were consistent 
with Neumann's (2003) categorization. Hard MOOCs represented expert knowledge and promoted 
mastery of concepts and professional skills. Soft MOOCs juxtaposed several viewpoints and promoted a 
critical stance towards learning. Note that the three hard MOOCs were introductory courses that lay the 
foundation for more advanced courses, which could also explain less emphasis on competing 
perspectives. 

Range of Assessment Methods 

Each MOOC incorporated a variety of graded and non-graded assessment methods ranging 
from multiple-choice questions to essays. Hard MOOCs used frequent non-graded assessment, such as 
in-video quizzes, with the majority of them being problems: 244 problems vs. 63 recall questions across 
three hard MOOCs. Soft MOOCs had 140 recall in-video quizzes. We noted a smaller number of non¬ 
recall in-video quizzes, and 13 comprehension questions in soft MOOCs. 

In this study, we identified an assessment method as formative if the outcome of the assessment 
informed teaching and learning. Learners in hard MOOCs had ample opportunity to use graded and 
non-graded assessments to master course material. The course logistics page of Programmingl MOOC 
indicated: “You can submit each exercise up to 5 times and we will use the highest grade from all of your 
submissions.” 

We also categorized assessment methods based on assessment mode. As Table 1 shows, peer- 
assessed and automatically-graded assessments formed almost all MOOC assessments for hard and soft 
MOOCs. Programming! and Stats MOOCs dedicated at least 20% of the final grade to peer- 
assessment. 

For soft MOOCs, the proportion of peer-assessed to automatically-graded assessment was 
different in each course. MentalHealth exclusively used peer-assessment while IntroPsych relied on 
automatic-grading. Instructor interview and curriculum documents, however, revealed that the 
IntroPsych MOOC employed a sophisticated form of automatic-grading implemented via a web-based 
tool that scaffolded active learning. For each question in this assessment tool, learners first answered the 
question in their own words. Then, they were shown four alternatives among which one was the correct 
answer. A wrong answer took learners to the relevant course material and allowed them to try the 
question a second time. 
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Table 1. Graded assessment in six MOOCs 


Assessment Mode (% of final grade) 


Disciplinary Category 

Title 

Peer-assessed 

Automatically-graded 

Hard 

Programmingl 


Final exam (25%) 

Three assignments (40%) 

Seven exercises (35%) 

Programming! 

Coding assignment (20%) 

Four exercises (60%) 

Assignment (20%) 


Stats 

Two assignments (30%) 

Seven exercises (70%) 


MentalHealth 

Three essays (100%) 

Bonus quiz (10%) 

Soft 

AboriginalEd 

Essay (50%) 

Two quizzes (40%) 

Discussion participation (10%) 


IntroPsych 

Essay (10%) 

Two web-based tests (90%) 


Peer-assessed components in all MOOCs had accompanying rubrics allowing the students to 
refine their answers before submitting their assignment. Learners in AboriginalEd, Stats, Programming!, 
and MetalHealth MOOCs only received full score for their peer-assessed assignments if they assessed 
their peers' submissions. Points were deducted for those learners who just submitted their own 
assignment. These rubrics also guided the peer-assessment process. Hard MOOCs included practice 
problems, while peer-assessed assignments in soft MOOC were essays. 

In one peer-reviewed assignment in MentalHealth MOOC students completed this assessment 
task: “Write a commentary on how the following terms reflect differences in the way societies have 
viewed mental health and illness over time: Lunatic asylum; Insane asylum; Mental Hospital; Mental 
health centre.” In the assignment, the instructor emphasized that “There is not a single correct answer to 
the questions-students should be evaluated based on their ability to apply relevant material from the 
course to their answer” (Curriculum documents, MentalHealth). 

To examine how assessment methods related to learning outcomes of each MOOC, we 
juxtaposed MOOC learning outcomes and assessment methods for each of the sixMOOCs. Assessment 
in hard MOOCs included problems and coding exercises, in the form of quizzes and assignments, while 
in soft MOOCs, learners had at least one assignment in the form of a short essay. Assignments and 
quizzes in the Stats MOOC addressed different learning outcomes: quizzes helped learners to master 
course content, while assignments that increased in complexity provided learners with an opportunity 
to, first, apply their knowledge and, eventually, interpret the results of a statistical analysis. The two 
Programming MOOCs included a variety of coding exercises that allowed learners to develop their 
coding skills, with Programming 2 MOOC providing a peer-assessment opportunity for learners that 
mirrored expected competencies in professional practice. The Programming! instructor further 
explained that code reviews: 

... are common in the workplace and before a piece of code would become part of the main 
system, it is very typical for that code to be reviewed by one of your colleagues. In a code review , 
you might give a suggestion for a different way) a simpler way of doing it or it might be a 
formality and say that you have checked all the boxes, you met the style guidelines, it looks the 
way it is expected to. (Programming 2, Instructor interview) 
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Three peer-reviewed short-essay assignments in MentalHealth MOOC enabled learners to 
apply the course content to their own interests and supported a reflective stance towards the social 
context of practice as relevant to mental health professionals. Assignments were open-ended, and the 
syllabus stated: “There is not a single correct answer to the questions-students should be evaluated 
based on their ability to apply relevant material from the course to their answer.” (Course Syllabus, 
MentalHealth). The last assignment specifically addressed one of the learning outcomes where learners 
evaluated contextual social factors that influenced mental illness and mental health. In AboriginalEd 
MOOC, one ethnographic essay, discussion in the course forum, and four non-graded reflective 
activities encouraged learners to evaluate available evidence and use it to construct arguments. For peer- 
assessment, learners were reminded to pay attention to how evidence supported a writer's argument 
rather than to judge how much the argument agreed with their own views. Argumentation and critical 
thinking skills in IntroPsych MOOC were fostered through a short essay. Learners could practice active 
learning and critical thinking while answering specially designed multiple-choice questions in mTuner. A 
sophisticated web-based approach to assessment, mTuner assessment tool embeds active learning in a 
multiple-choice format. mTuner is conceptualized and implemented by the instructor's research lab to 
encourage actively engaging with learning while being assessed. Here is a description of mTuner 
obtained from the “Active Learning: Online Redesign” project website 
(http: / / alor.onlinelearning.utoronto.ca/web-application-development): 

m Tuner assessment tool incorporates all of the existing research on “assessment FOR learning" 
to provide the most powerfullearning context possible. Students are never more engaged than 
when their learning is being assessed. This assessment tool asks students to first generate 
answers, it supports the deepest learning experience, informs students when their answers are 
correct to reinforce accurate learning provides students with hints and gives them a second 
chance when they initially are incorrect. It corrects misconceptions, and by summarizing the 
correct answer to every question before moving on, it deepens the learning experience. 

For each mTuner question, learners first attempted to answer the question in their own words. 
Then, they were shown four alternatives among which one was the correct answer. Upon choosing the 
correct answer, learners were presented with an explanation about that answer. A wrong answer took 
learners to the relevant course material and allowed them to try the question a second time. 

Interaction Design 

We focused on students and instructors’ interaction on the platform-provided discussion tool. 
IntroPsych MOOC included external components, for example, that were heavily reliant on peer 
interaction and collaboration. 

Purpose of interaction. As extracted from course documents and instructor interviews, student- 
student and student-instructor interaction served numerous purposes (see Table 2). Note that 
examining the content of the discussion forum posts was out of the scope of this particular study but will 
be considered for future analysis. 
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Table 2. Purposes of interaction in the six MOOCs 


Purpose of Interaction 


Q&A 

Discussion 

Diversity Empowerment 

Community 

Feedback to 

Instructor 

Programming 1 

/ 




/ 


Hard 







MOOC, P ™S»mmi„ g 2 

/ 




/ 


Stats 

/ 




/ 


MentalHeath 

/ 

/ 

/ 


/ 

/ 

Soft .. 

MOOCs Ab0riglna ' Ed 

/ 

/ 

/ 

/ 

/ 


IntroPsych 

/ 

/ 

/ 


/ 

/ 


Asking and answering questions was a common purpose of discussion forum use in Soft and 
Hard MOOCs. The other common purpose was community building, which, in the case of the 
MentalHealth MOOC allowed different stakeholders to bring various perspectives to the conversations. 
For Soft MOOCs, discussion and idea sharing were indicated as other reasons for participation. 
Exposure to diversity, nevertheless, was unique to Soft MOOCs. Such emphasis links back to the 
curricular content of soft disciplines, validating the diversity of ideas and perspectives. Empowering 
learners to take action by engaging them in discourse was the ultimate rationale for discussion 
participation in the AboriginalEd MOOC. 

Regardless of the relevance of students' discussion forum participation to their learning, course 
announcements, curricular documents, and instructor interviews showed that peer interaction was 
valued and encouraged in all six MOOCs. Consider the following announcement shared with 
MentalHealth learners as one example: ”We hope that you will take the time to not only watch the 
lectures but also participate in the forums by asking and answering questions and sharing your 
thoughts.” (Course announcement) 

Choice of discussion topics, learning activities that encouraged learners to share their 
perspectives with each other, and providing questions that instigated conversation among learners were 
recognized by the MentalHealth and the IntroPsych instructors as strategies to increase peer interaction 
and learner engagement in course discussion. Reflecting on ways to encourage learner participation in 
discussions and to promote deeper thinking, the Stats instructor entertained the potential of integrating 
discussion prompts so that learners could exchange ideas about concepts covered in a given week of the 
course beyond the scope of exercises. 

Discussion forum structure. 

Overall, discussion forum structure across MOOCs mirrored course content, activities, and 
assignments with each component of the course being assigned a sub-forum. Such structure facilitated 
learners posting their questions and comments to an appropriate sub-forum, which in turn, allowed the 
instructors to monitor ongoing discussions more efficiently. Meta sub-forums and threads created by the 
instructors sometimes moved beyond course components to foster deeper discussions. In the 
IntroPsych MOOC, for instance, a member of the instructional team initiated a Summary thread and a 
Question of the Week thread in the course sub-forums. 
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A common issue brought up by the MOOC instructors was that learners posted similar 
questions or comments in different threads, which hindered the instructional teams' efforts to effectively 
monitor the discourse and address questions or comments. Having faced this problem in the first 
offering of Programming 1 where learners created threads with repetitive and/or overlapping content, 
the instructors implemented a highly structured discussion forum design in Programming! and in the 
second offering of Programmingl and only monitored posts that were created in an appropriate sub¬ 
forum. The instructors described their discussion forum design rationale: 

Instructor 1: In the first one > ...we had these discussion groups per assignment and per exercise... 
And that turned out to be so free form that people posted the same question literally dozens of 
times. And it just overwhelmed us with questions and we eventually gave up in trying to keep up 
with all the different areas and we focused on the ones that were posted in the right place. And 
then Instructor2 decided that we were going to change. 

Instructor2: The goal the second time around was to make it clear exactly which sub-forum you 
would post a question in... It was all very targeted and for something like the exercises we had 
sub-forums by question... You had to drill down but when you got there > you could more easily 
spot the question you were asking. (Instructors’interview, Programmingl&2) 

Learners could deviate from the initial discussion forum structure in IntroPsych, AboriginalEd, 
and MentalHealth MOOCs. For instance, students in the MentalHealth course could respond to weekly 
questions posted by the instructor or according to the Course Syllabus “start discussions using their own 
questions.” The instructor encouraged learners’ unprompted contributions to the discussion forum in a 
message shared in the second week of the course: “I had put in some special questions but from what I 
can tell, you guys don't need my help, there are terrific discussions happening” (Prologue to Lecture 2, 
MentalHealth). 

Instructional presence in course discussions. 

While the quantity of forum posts appeared overwhelming and monitoring the threads was time 
consuming, instructors described their strategies for monitoring the discussion forum and commented 
when needed. Only one instructor tackled the discussion forums alone. The other instructors formed an 
instructional team with designated assistants who scanned discussion forum posts along with the 
instructor, answered questions when needed, or let the instructor know if direct intervention was 
required. 

Monitoring the course discussion forum served three purposes: answering questions, checking 
the relevance and accuracy of content contributed to the forums, and detecting potential deviances from 
discussion forum code of conduct. In the second offering of Programmingl, a number of learners who 
had completed the first offering were invited to join the instructors as Community Teaching Assistants 
(CTAs). According to the instructors, CTAs effectively monitored discussion threads and answered 
questions. Programming MOOC instructors also focused their attention to sub-forums, such as 
assignments, where multiple students could benefit from their replies to questions: “The areas that we 
monitored were heavily trafficked but we also had a lot of bang for the buck where answering a question 
that was confusing to a lot of people had a big impact.” (instructors’ interview, Programmingl& 2). 

In their interviews, instructors reflected on how their level of involvement in course discussions 
impacted learners' participation. The AboriginalEd MOOC instructor kept an active presence in 
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discussion forums following Programmingl&2 MOOC instructors' advice, and believed that learners 
would become more engaged in discussions knowing that the instructor was actually following the 
conversation. Only one MOOC instructor, Stats, speculated that their presence in the discussion forum 
might have shortened the threads as learners would see the correct answer and not continue the 
discussion. 

Beyond commenting in discussion threads, MOOC instructors sometimes intervened to 
address conflicts. Instances of conflict instigated by controversial topics existed in all three Soft 
MOOCs. Instructors addressed content-related conflicts by commenting in relevant threads to provide 
further information, creating response videos, or sending messages to learners through course 
announcements. After discussions regarding a controversial topic led to a heated argument, in addition 
to providing a commentary, the IntroPsych MOOC instructor communicated the following message to 
learners: 

Ipersonally don't want to dwell on this issue too long; there are too many other interesting 
things to be discussing as well\ but I don't want to blow it off either. Pondering how we came to 
be is a big part of the human experience; and I do think about these issues often. So it's worth 
discussion, in a respectful way with open minds and big ears. (Announcement; IntroPsych) 

DISCUSSION AND IMPLICATIONS 

We examined course design in six U of T MOOCs using an epistemologically informed 
framework (Neumann et. ah, 2002) to guide our analysis along two broad disciplinary categories of hard 
and soft. We identified differences and similarities across four dimensions in MOOCs belonging to 
different disciplines: learning outcomes, curriculum and content, assessment, and interaction design. 
Findings of our study demonstrated that MOOC instructors designed their courses with the intention of 
actively engaging MOOC learners within the inherent technical constraints of MOOC environments, a 
situational factor beyond their course design decision-making. With the underpinning universal course 
template, the six courses appeared unvaried in high level design and structure. However, instructors of 
each course adapted the existing tools of the Coursera platform as much as possible to foster desired 
learning outcomes for their diverse learners. When possible, as in the case of IntroPsych course, other 
tools were integrated into the platform to enrich the learning experience. 

MOOC Learning Outcomes Related to Disciplinary Expectations 

Learning outcomes of the sixMOOCs reflected broad cognitive competencies that each course 
promoted. Soft MOOCs explicitly emphasized the importance of critical thinking, evidence-based 
argumentation, and evaluation of evidence. Developing knowledge of fundamental concepts and 
methods of a field and, subsequently, applying that knowledge in solving problems, characterized the 
learning outcomes of hard MOOCs. The emphasis on critical thinking skills and argumentation in soft 
fields, and on deep knowledge of principles and being skilful in the application of methods in hard 
disciplines, are highlighted in conceptual papers (Neumann et al., 2002) and studies that examined 
instructors’ views of discipline-related learning goals in undergraduate education (Hativa, 1997). 

While differences in high level learning outcomes existed between hard and soft MOOCs, we 
observed commonalities with respect to cognitive skills with broad applications. The Stats MOOC 
instructor, a hard course, noted the importance of critical thinking, which, in higher education is 
recognized as a trans-disciplinary skill (Krause, 2014). 
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Content and Curriculum 

Curricular content and instructor’s beliefs about disciplinary ways of knowing and methods for 
authenticating knowledge assertions, revealed differences between soft MOOCs—IntroPsych, 
AboriginalEd, and MentalHealth—and hard MOOCs—Stats, Programmingl, and Programming!. 
Synthesizing existing conceptions of teaching in different disciplines (Neumann et ah, 2003), courses 
associated with hard disciplines tend to have structured and sequenced content. In soft disciplines, 
competing paradigms often coexist, leading to courses that have less structured content and showcase 
more of a spiral curriculum. Soft MOOCs instructors all communicated with learners the importance of 
context with respect to content. However, the three hard MOOCs were introductory courses where 
content knowledge is established, so that also influenced design decisions. Understanding the 
curriculum and content design in more advanced hard MOOCs, in a similar MOOC platform, is a 
compelling line of future research. 

Student-centeredness, within the curricular context of a course, was another characteristic of 
curricular content in soft MOOCs. All three soft MOOC instructors, as explained in the findings section, 
created new content in response to discussions around existing curricular content. In these courses, 
students actively shaped course content and built on the foundation of the instructors’ original visions. 

Course content in this study was examined from a discipline specific epistemological 
perspective, which differentiates our findings from the work of Swan et al. (2014), where content was 
rated on a continuum of abstract to concrete regardless of a course’s disciplinary affiliation. Swan et al. 
(2014) also defined structured-ness of a MOOC in terms of consistent organization and presentation of 
content. Using such criteria, we could not differentiate among the hard and soft MOOCs, as they would 
differ only minimally. Instead, informed by our conceptual framework, we interpreted structured-ness as 
the degree to which the original course content reflected the characteristics of disciplinary knowledge 
(Donald, 1995) and yielded itself to tangential discussion and opportunistic expansion. Building on the 
learning outcomes of their MOOCs, instructors in this study constructed their course content in 
accordance with disciplinary expectations. 

Assessment Methods 

All six MOOCs utilized graded and non-graded assessment methods. For hard MOOCs, with 
more emphasis on mastery of methods and application of knowledge, practice problems comprised the 
bulk of non-graded assessments. Hard MOOCs also extended the opportunity for formative assessment 
to some of their quizzes and assignments, where learners could submit their answers multiple times. 
Jessop and Malekar (2016) have reported similar findings with respect to the use of formative 
assessment in science courses. 

The type and number of graded assessments in these sixMOOCs reflected Neumann et al’s 
(2002) characterization of assessment in different disciplinary categories. Soft MOOCs had fewer 
graded assessments compared to hard MOOCs. Assessment methods in hard MOOCs were 
quantitative, e.g. mathematical problems and coding assignments. For soft MOOCs, specifically 
AboriginalEd and MentalHealth, short essays encouraged learners to craft evidence-based and 
contextualized arguments. With formative assessment far more explicitly evident in hard MOOCs, we 
propose that soft MOOC instructors consider ways to include scaffolded opportunities for formative 
assessment where learners can self-assess their work and self-diagnose areas of improvement. 

An unexpected observation was that 90% of students’ final grade in IntroPsych, a soft MOOC, 
was implemented as automatically-graded questions. An examination of course assessment, however, 
revealed that one of the non-graded course activities, implemented as an external web-based tool, 
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utilized a technological system that allowed learners to experience an authentic research cycle. Another 
assessment design feature in the IntroPsych MOOC was a web-based assessment tool that embedded 
active learning in a seemingly multiple-choice format. In MOOC-like learning contexts, where the ratio 
of learners to instructional team members does not allow frequent personalized feedback, such 
assessment tools that add a formative element to standard multiple-choice questions can engage learners 
in monitoring and recalibrating their understanding of course material. 

Peer-graded assignments were included in five MOOCs. Learners received detailed instructions 
on the process of peer-assessment and were also given grading rubrics and guidelines. Some of the 
MOOC instructors used strategies to encourage learners to actively engage in peer-assessment. For 
example, a full score for a peer-graded assignment was only awarded if a learner assessed the work of 
their peers. 

Peer-assessment is a fertile ground for research in MOOCs, or in large enrollment classes in 
general (Pare, et al, 2015). Researchers have, for example, conducted empirical studies to examine the 
implications of rubric construction on completion rate and quality of peer feedback (Yousef, Wahid, 
Chatti, Schroeder, & Wosnitza, 2015). We acknowledge that without analyzing actual peer assessment 
data in this current study, we cannot comment on the effectiveness of such motivational strategies. 

Assessment methods in each MOOC aligned well with their identified learning outcomes. While 
none of the MOOC instructors indicated following a specific model, such as backward design (Wiggins 
& McTighe, 2005), to establish such alignment, we posit that MOOC instructional design training and 
support in this particular university stressed the importance of using learning outcomes as the 
foundation upon which instructional activities and assessment methods are built. 

Interaction Design and Instructor Involvement 

Participation in discussion forums was encouraged in the six MOOCs, even though interactions 
served different purposes in hard and soft MOOCs: soft MOOCs emphasized discussion and exposure 
to a variety of contexts and perspectives while hard MOOCs had more emphasis on question and answer 
type interactions. Aggregate numbers of posts in the discussion forum of the MOOCs included in this 
study are reported in “Open U of Toronto MOOC Initiative” reports (2014). However, calculating the 
amount of interaction corresponding to each purpose may be carried out via content analysis in a future 
study. 

Regarding strategies for increasing participation in the discussion forums, instructors used 
course announcements to encourage learner participation. The six MOOCs in this study did not 
incorporate deliberate treatments, such as customized messages sent to experimental learner groups. Yet 
lack of such treatments does not translate into design flaw or drawbacks. The effectiveness of targeted 
treatment to enhance participation in MOOCs is far from established. For example, in two studies, 
Kizilcec, Schneider, Cohen, and McFarland (2014) observed an adverse impact for persuasive emails 
sent to learners where learners who received neutral emails reminding them of the existence of a 
discussion forum, participated in higher rates compared to learners who received persuasive emails that 
highlighted the importance of individual or collective learning achieved through participation in 
discussions. 

The level of instructors' presence in discussion forums and its impact on the length of discussion 

threads was a concern for one of the instructors, reflecting the findings of Mazzolini and Maddison 

(2007). As well, Tomkin and Charlevoix (2014) investigated the role of MOOC instructor presence on 

learner engagement in discussion forums in an Environmental Science MOOC where learners were 

assigned randomly to two groups: a control group with no instructional presence, and a treatment group 
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with instructional presence in discussion threads. The treatment group and the control group did not 
differ in participation rates in course discussions. In our study, the total number of posts in course 
discussion forums varied between 5,000 and 31,000 in all six MOOCs. Yet a clear distinction between 
the number of posts in hard and soft MOOCs was not evident. Programmingl, a hard MOOC, had 
more than 25,000 posts while learners in MentalHealth contributed a total number of 11,700 discussion 
posts. 

Beyond demonstrating higher number of learners in soft MOOCs who posted at least once in 
any discussion thread, our numerical findings from discussion forum data were inconclusive in 
supporting a distinction between hard and soft MOOC with regards to students and instructors’ 
participation and contribution to MOOC discussion threads. Specifically, contrary to Mazzolini and 
Madison’s (2007) study, we found that median length of threads with any instructor involvement was 
higher in all MOOCs than threads with no instructor involvement. Further research, such as a study 
underway by Chandrasekaran et al. (2015), is needed to investigate how instructional presence in 
MOOC discussion forums can be supported to have more positive learner impact. 

SIGNIFICANCE 

To explore variations in MOOC design we mapped the design of teaching and learning 
components of MOOCs from various disciplines to a widely-used framework for understanding 
potential differences and similarities across MOOCs. We emphasize, again, that Neuman et al.'s (2002) 
framework provided an analytical lens through which we recognized main dimensions of teaching and 
learning rather than prescribing our search for expected differences or commonalities. Few studies have 
formally examined pedagogical design considerations of MOOCs (Swan et al., 2014). Studies of 
instructional presence within a MOOC context (Chandrasekaran et al., 2015), or comparing learner 
engagement in MOOCs differing in their level and topic (Cofrin, Corrin, Barba, & Kennedy, 2014) are, 
at the time of reporting this study, starting to emerge in MOOC literature. With the awareness of how 
pedagogical components of MOOCs may vary from course to course and may reflect underlying 
disciplinary assumptions, our next step is to examine interaction design in MOOCs to probe the scope 
and depth of student-student and student-instructor interaction. 

At least one instructor in this study raised concerns about the lack of pedagogical models for 
MOOC design; for example, interaction models that promote learning. Successful integration of active 
learning in high-enrolment undergraduate courses (Drinkwater et al., 2014) could provide MOOC 
instructors with design ideas. In the present study, alternative assessment approaches that were 
integrated in the IntroPsych MOOC, for example, show that there is potential for fostering active 
learning in large and diverse learning environments similar to MOOCs. 

One implication of this research is to propose opportunities for fostering active learning and 
deep, topic-related discussions by adopting novel pedagogical approaches informed by practices of the 
extant disciplines or adapted from successful models in other disciplines. In this paper, we took one step 
forward in deconstructing curriculum design and implementation in six MOOCs representing a range of 
disciplinary affiliation to identify instances of such practices. 

MOOC instructors who participated in this study attempted to foster ways of knowing, relevant 
perspectives, professional practice, fundamental skills, and foundational knowledge in their subject 
domain and within the parameters of a generic MOOC platform. Several indications of promising 
practice have emerged from this study. First, MOOCs can be a sand box for testing innovative 
pedagogical practice (Fischer & Wolf, 2015) to further inform teaching and learning in on-campus 
courses. Second, the need for scalability may serve as a catalyst for instructors to explore design 
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strategies outside of their disciplinary traditions. Finally, the work of MOOC instructors at the 
University of Toronto shows early signals of the value of exposure to pedagogical approaches of 
instructors from other subject areas. These instructors have had access to both formal and informal 
professional development activities that facilitate sharing of experience of both the process and 
outcomes (e.g., the role of peer-assessment in fostering critical thinking). Creating further opportunities 
for exchange of ideas both within and across discipline areas may serve to expand the repertoire of 
approaches to innovative curricular design. 
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